[SPARK-32034][SQL] Port HIVE-14817: Shutdown the SessionManager timeoutChecker thread properly upon shutdown by yaooqinn · Pull Request #28870 · apache/spark

yaooqinn · 2020-06-19T08:54:00Z

What changes were proposed in this pull request?

This PR port https://issues.apache.org/jira/browse/HIVE-14817 for spark thrift server.

Why are the changes needed?

When stopping the HiveServer2, the non-daemon thread stops the server from terminating

"HiveServer2-Background-Pool: Thread-79" #79 prio=5 os_prio=31 tid=0x00007fde26138800 nid=0x13713 waiting on condition [0x0000700010c32000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
	at java.lang.Thread.sleep(Native Method)
	at org.apache.hive.service.cli.session.SessionManager$1.sleepInterval(SessionManager.java:178)
	at org.apache.hive.service.cli.session.SessionManager$1.run(SessionManager.java:156)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

Here is an example to reproduce:
https://github.com/yaooqinn/kyuubi/blob/master/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/spark/SparkSQLEngineApp.scala

Also, it causes issues as HIVE-14817 described which

Does this PR introduce any user-facing change?

NO

How was this patch tested?

Passing Jenkins

…utChecker thread properly upon shutdown

SparkQA · 2020-06-19T09:26:50Z

Test build #124277 has finished for PR 28870 at commit 1907832.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

wangyum

Do we need to backport to hive-1.2?

yaooqinn · 2020-06-19T17:51:49Z

retest this please

SparkQA · 2020-06-19T18:22:34Z

Test build #124292 has finished for PR 28870 at commit 756e084.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-06-19T18:28:53Z

Test build #124293 has finished for PR 28870 at commit 756e084.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2020-06-19T22:10:59Z

...hive-thriftserver/v1.2/src/main/java/org/apache/hive/service/cli/session/SessionManager.java

  @Override
  public synchronized void stop() {
    super.stop();
    shutdown = true;


This should be removed because we did it in shutdownTimeoutChecker function.
HIVE-14817 removed this, too.

dongjoon-hyun · 2020-06-19T22:11:13Z

...hive-thriftserver/v2.3/src/main/java/org/apache/hive/service/cli/session/SessionManager.java

  @Override
  public synchronized void stop() {
    super.stop();
    shutdown = true;


This should be removed because we did it in shutdownTimeoutChecker function.
HIVE-14817 removed this, too.

dongjoon-hyun · 2020-06-20T22:02:16Z

Gentle ping, @yaooqinn . :)

yaooqinn · 2020-06-21T04:53:40Z

@dongjoon-hyun thanks for the check!

SparkQA · 2020-06-21T05:28:59Z

Test build #124331 has finished for PR 28870 at commit 68a0c7b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun

+1, LGTM. Thank you all.
Merged to master/3.0.

dongjoon-hyun · 2020-06-21T23:29:11Z

BTW, @yaooqinn . Is this relevant to branch-2.4, too?

…utChecker thread properly upon shutdown ### What changes were proposed in this pull request? This PR port https://issues.apache.org/jira/browse/HIVE-14817 for spark thrift server. ### Why are the changes needed? When stopping the HiveServer2, the non-daemon thread stops the server from terminating ```sql "HiveServer2-Background-Pool: Thread-79" #79 prio=5 os_prio=31 tid=0x00007fde26138800 nid=0x13713 waiting on condition [0x0000700010c32000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) at org.apache.hive.service.cli.session.SessionManager$1.sleepInterval(SessionManager.java:178) at org.apache.hive.service.cli.session.SessionManager$1.run(SessionManager.java:156) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) ``` Here is an example to reproduce: https://github.com/yaooqinn/kyuubi/blob/master/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/spark/SparkSQLEngineApp.scala Also, it causes issues as HIVE-14817 described which ### Does this PR introduce _any_ user-facing change? NO ### How was this patch tested? Passing Jenkins Closes #28870 from yaooqinn/SPARK-32034. Authored-by: Kent Yao <yaooqinn@hotmail.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> (cherry picked from commit 9f8e15b) Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

yaooqinn · 2020-06-22T02:34:24Z

BTW, @yaooqinn . Is this relevant to branch-2.4, too?

yes. need a followup?

dongjoon-hyun · 2020-06-22T02:44:30Z

If you don't mind, could you make a backporting PR then?

yaooqinn · 2020-06-22T02:48:44Z

OK

dongjoon-hyun · 2020-06-22T02:50:03Z

Thank you!

…timeoutChecker thread properly upon shutdown ### What changes were proposed in this pull request? This PR backports #28870 which ports https://issues.apache.org/jira/browse/HIVE-14817 for spark thrift server. ### Why are the changes needed? Port HIVE-14817 to fix related issues ### Does this PR introduce _any_ user-facing change? no ### How was this patch tested? passing Jenkins Closes #28888 from yaooqinn/SPARK-32034-24. Authored-by: Kent Yao <yaooqinn@hotmail.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

gatorsmile · 2020-06-24T02:15:29Z

cc @juliuszsompolski

juliuszsompolski · 2020-06-24T12:21:59Z

LGTM, thanks!

[SPARK-32034][SQL] Port HIVE-14817: Shutdown the SessionManager timeo…

1907832

…utChecker thread properly upon shutdown

probot-autolabeler bot added the SQL label Jun 19, 2020

wangyum reviewed Jun 19, 2020

View reviewed changes

hive1.2

756e084

yaooqinn changed the title ~~[SPARK-32034][SQL] Port HIVE-14817: Shutdown the SessionManager timeoutChecker thread properly upon shutdown~~ [SPARK-32034][SQL][test-hive1.2] Port HIVE-14817: Shutdown the SessionManager timeoutChecker thread properly upon shutdown Jun 19, 2020

dongjoon-hyun reviewed Jun 19, 2020

View reviewed changes

fix

68a0c7b

dongjoon-hyun changed the title ~~[SPARK-32034][SQL][test-hive1.2] Port HIVE-14817: Shutdown the SessionManager timeoutChecker thread properly upon shutdown~~ [SPARK-32034][SQL] Port HIVE-14817: Shutdown the SessionManager timeoutChecker thread properly upon shutdown Jun 21, 2020

dongjoon-hyun approved these changes Jun 21, 2020

View reviewed changes

dongjoon-hyun closed this in 9f8e15b Jun 21, 2020

yaooqinn mentioned this pull request Jun 22, 2020

[SPARK-32034][SQL][2.4] Port HIVE-14817: Shutdown the SessionManager timeoutChecker thread properly upon shutdown #28888

Closed

Conversation

yaooqinn commented Jun 19, 2020

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

SparkQA commented Jun 19, 2020

Uh oh!

wangyum left a comment

Choose a reason for hiding this comment

Uh oh!

yaooqinn commented Jun 19, 2020

Uh oh!

SparkQA commented Jun 19, 2020

Uh oh!

SparkQA commented Jun 19, 2020

Uh oh!

dongjoon-hyun Jun 19, 2020

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun Jun 19, 2020

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented Jun 20, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yaooqinn commented Jun 21, 2020

Uh oh!

SparkQA commented Jun 21, 2020

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented Jun 21, 2020

Uh oh!

yaooqinn commented Jun 22, 2020

Uh oh!

dongjoon-hyun commented Jun 22, 2020

Uh oh!

yaooqinn commented Jun 22, 2020

Uh oh!

dongjoon-hyun commented Jun 22, 2020

Uh oh!

gatorsmile commented Jun 24, 2020

Uh oh!

juliuszsompolski commented Jun 24, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

dongjoon-hyun commented Jun 20, 2020 •

edited

Loading